Most single-chip digital image sensors used in digital cameras, camcorders, and scanners to create a color image use a Bayer filter mosaic. This type of color filter array, as described in U.S. Pat. No. 3,971,065 which is hereby incorporated by reference, includes a square grid of photosensors covered with red, green, and blue filters. The filter pattern is 50% green, 25% red and 25% blue. Rows of red and green filters alternate with rows of blue and green filters, with the green filters in different rows offset by one column.
Image sensors (e.g. CCDs and CMOS sensors) normally support modes of reduced resolution in order to achieve higher frame rates. When operating in these modes, the Bayer filter image will undergo downscaling before it is read out from the sensor. The operation of Bayer downscaling performed in the sensor is often called “sensor binning”, and is performed by summing (or averaging) several input photosensor signals into one output pixel. Sensor binning is commonly applied during movie capture, where high frame rates are required, albeit at the expense of resolution.
When performing image downscaling, it is usually important to maintain the property that neighboring image pixels are equally spaced from each other. Pixel locations complying with this property are referred to as being “ideal” because they lie on a uniform grid. One common side effect of sensor binning is “channel displacement”, meaning that the pixels of the downscaled image are spatially displaced from their ideal locations. This displacement is different for each of the four Bayer channels: R, Gr, Gb, B (where Gr refers to green pixels in a line containing red pixels, Gb refers to green pixels in a line containing blue pixels). In other words, the red component of the downscaled image is displaced in one direction, the Gr component of the downscaled image is displaced in another direction, and so on. The result is that the property of equal spacing between neighboring pixels is violated in the down-scaled image.
For example, let p be the spacing between one pixel to its neighboring pixel on one side, and let q be the spacing between the same pixel to its neighboring pixel on the opposite side. The “channel displacement ratio” is then defined as p/q. If the channel displacement ratio is equal to one, then there is no channel displacement, and the property of equal spacing between neighboring pixels is maintained. If the ratio is different than one, then there is channel displacement, and the property of equal spacing between neighboring pixels is violated. Channel displacement ratios may be different in the horizontal and vertical directions, or may be the same.
The side effects of channel displacement include jagged diagonal edges and false colors along edges. The present invention is directed to reducing or eliminating these undesirable artifacts.
Referring now to FIG. 1, an example is presented for conventional sensor binning, in this case reducing image resolution by factor of 1:2 in each direction (horizontal and vertical). According to this particular example, each 4×4 pixel unit is reduced to a 2×2 pixel unit in the following manner:                All four red pixels within the 4×4 pixel unit are summed (or averaged) in order to produce one red pixel. Since all four red pixels were summed (or averaged) with the same weight, the output red pixel is located exactly in the middle of the four red pixels.        All four Gr pixels within the 4×4 pixel unit are summed (or averaged) in order to produce one Gr pixel. Since all four Gr pixels were summed (or averaged) with the same weight, the output Gr pixel is located exactly in the middle of the four Gr pixels.        All four Gb pixels within the 4×4 pixel unit are summed (or averaged) in order to produce one Gb pixel. Since all four Gb pixels were summed (or averaged) with the same weight, the output Gb pixel is located exactly in the middle of the four Gb pixels.        All four blue pixels within the 4×4 pixel unit are summed (or averaged) in order to produce one blue pixel. Since all four blue pixels were summed (or averaged) with the same weight, the output blue pixel is located exactly in the middle of the four blue pixels.        
When comparing FIG. 1 image (b) to image (c), presenting the ideal location of the pixels, it is clear that in image (b) the red component of the downscaled image is displaced in the bottom-right direction from its ideal location. The Gr component of the downscaled image is displaced in the bottom-left direction from its ideal location. The Gb component of the downscaled image is displaced in the top-right direction from its ideal location. And the blue component of the downscaled image is displaced in the top-left direction from its ideal location.
Also, in image (b) of FIG. 1, there is not equal spacing between neighboring pixels. The spacing between red pixels and neighboring green pixels on the right is one [opu] (opu is “Original Pixel Unit”). However, the spacing between red pixels and neighboring green pixels on the left is three [opu]. This means that the channel displacement ratio in the horizontal direction is 1:3.
In the particular example presented in FIG. 1, sensor binning in the vertical direction is done exactly the same as in the horizontal direction, and therefore channel displacement ratio is 1:3 in both directions. However, this is not necessarily so. Some sensor binning schemes may invoke different channel displacement ratios in the horizontal direction and in the vertical direction.
The channel displacement side effect of sensor binning is currently handled in the art in two basic ways:
1. Post Demosaicing solutions—this option includes filters operating in the RGB/YUV domain, for example:
a. Applying a chroma low pass filter or a chroma median filter to remove the false colors caused by channel displacement.
b. Applying an edge preserving luma low pass filter, in order to smooth the diagonal luma edges (as mentioned earlier, channel displacement causes jagged diagonal edges).
Post demosaicing solutions may solve the problem to some extent, but they may also cause a lot of damage to the image, such as excessive edge smoothing (loss of sharpness), chroma blur, discontinuities, etc. One of the reasons is that demosaicing normally enhances the side effects of channel displacement, and in a way that makes it more difficult to correct without damaging the image.
2. Pre Demosaicing solutions—these include applying a diagonal edge smoothing filter on the green channel, similar to the edge preserving luma low pass filter mentioned before. However, the common pre-demosaicing solution for correcting the channel displacement side effects of sensor binning is to apply some operator on the Bayer image that will shift the pixels to their ideal location, by means of interpolation (see FIG. 1, image (c)).
Normally, bi-linear interpolation will be applied separately on each of the four components of the image (R, Gr, Gb, and B). However, bi-linear interpolation has the side effect of blurring the image. Alternatively, it is possible to apply bi-cubic interpolation separately on each of the four components of the image. This will produce a sharper image. However, applying bi-cubic interpolation separately on each channel may require a big filter and heavy computation. Also, bi-cubic interpolation results will still not be sufficiently accurate, and artifacts may still occur. This is mainly due to the fact that in both methods (bi-linear and bi-cubic) the Gb pixels are not considered when shifting the Gr pixels, and the Gr pixels are not considered when shifting the Gb pixels.